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FOREWORD 



The Computer-Based Instructional Systems Team of the US Army Research 
Institute for the Behavioral and Social Sciences (ARI) performs research 
and development in the area of educational technology that applies to 
military training. Of interest are methods for training individuals to 
develop and utilize instructional courseware in reasonable time, at acceptable 
cost. * 

This Research Product is one of a series which have been designed to 
support the implementation of the Instfuctonal Systems Development Model 
(ISD, TRADOC Pamphlet 350-30). The ISD Model is a st£p-by-step procedure 
for the analysis, design, development, implementation, and control of mili- 
tary course materials. A previous effort produced manual Job Aids which are 
paper and pencil documents designed to provide "how to do it 11 guidance for 
the ISD Model. This document is part of a series of three developed to 
support the delivery of the manual Job Aids by computer. To accomplish this 
research, ARI ! s ^resources were augmented by contract DAHC19-78-C-0100 with 
the Human Resources Research Organization (HumRRO). ^ 

t 

The contributions of personnel from ARI f s Manpower and" Educational 
SysteiflsJs Technical Area as well as those of Mr. Charges F. ( Marshall and Mr. 
Joseph P. Severo, Research* Facilities Support Group are acknowledged. Mr. 
Antonio J. Alameda, HumRRO also contributed" to this research effort. 

The entire research project is responsive to ths requirements of Army 
Project 2Q263743A794, FY80 Work Program. 

<J0EEP& Z eTt Tm E R - 
T£<^hnica\L Director 
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SUPPLEMENTAL GUIDE: SOURCES OF INFORMATION FOR ON-LINE IMPLEMENTATION OF 
ISD T.5 SELECT INSTRUCTIONAL SETTING . c 'I i ' ' 



•BRIEF 



<< 



a 



Requirement: t 9 

The purpose was to develop a language to translate an existing paper and 
pencil Job Aid onto a computerized delivery system. The , Job Aid is one 'of 
series developed 'previously to support users of the Instructional Systems 
Development Model (ISD). » 



Procedure : : . / - 

i 

A Programming Design Language (PDL) was created to describe the computer 
functions (e.g., computer /user interactions, storage/retrieval of data, program 
branching, program management , and calculations) required by the Job Aids (ARI 
Research, Products 80-13 through 80-18). s The PDL was designed to communicate 
to the computer programmer in a language independent fashion so that the on-line 
or computer version of the Job Aid could be delivered by any computer^ This 
document was designed to support the user of the computer-based Job Aid. 



Utilization: 

# 

The Programming Design Guide may. be used by computer programmers who are 
tasked with programming % the manual Job- Aids.' This supplemental guide is 
intended for. use by Instructional Developers 'who may use the on-line Job Aid. 
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Chapter I, 



INTRODUCTION 



Job aids are being developed, for the U.S. Army Research Institute for the Behavioral 
and Social Sciences under contract DAHC19-78-£-0Q10. The job aids are intended to be 
stand-alone, step-by-step procedural guides which are equally useful to individuals at all 
experience levels of the instructional systems development process. 

{There are thirteen job aids presently available. t They are: 



* • 



Job Aid for Selecting Tasks for Training * 

Job Aid for Conducting Task Analysis * 

Job Aid for Analyzing Existing Courses * 

Job Aid for Selecting Instructional Setting 

Job Aid for Developing Objectives ( • 
Job Aid for Developing Tests x ' * \ • 

Job Aid for Describing Entry Behavior « 
Job Aid for Determining Sequence and Structure - * 

Job Aid for Specifying Learning Events and Activities 

Job Aid for Specifying Instructional Management Plan and Delivery System 

Job Aid for Review and Selection of Exiting Materials 

Job Aid for Developing Instruction , 

Job Aid for Validating Instruction 



In addition to printed versions of the above job aids, the Job Aid for Selecting 
Instructional Setting can be made availably in a computer version. A Programming Design 
Guide 1 is available for use by computer programmers which provides all of the guidance 
necessary for them to implement the job aid on their computer system. Instructional 
development personnel would then use. the on-line versioa.of the aid to assist in the selec- 
tion of instructional setting for each critical task. This document provides^supplemental 
guidance for the user jof the on-line job aid. It is no\ intended to be a stand-alone 
publication. « . t 
V : : 



THJS SUPPLEMENTAL GUIDE IS NOT INTENDED TO BE A 
STAND-ALONE DOCUMENT. IT IS IN.TENDEDTO PROVIDE 
ONLY SUPPLEMENTAL INFORMATION JHAT MAY BE USEFUL* 
TO INSTRUCTIONAL DEVELOPMENT PERSONNEL WHO USE 
THE ON-LINE VERSION OF THE JOB AID SELECT INSTRUC- 
TIONAL SETTING. - . 



0 1 — : — 

^chulz, R.ES, Underhill, W.G., Hargan, C.S. Programming Design Guide for Computer Imple- 
mentation of Job Aid for Selecting Instructional Settings, (Research Product 80-24b). Alexandria, VA. 
US Army Research Institute, August 1980. * g 
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Chapter II 

/. -.n SOURCES OF INFORMATION 

OVERVIEW OF THE INSTRUCTIONAL SETTINGS JOB AID 

• Objective • 

1. Given "a list of tasks selected for training'within a single skill level, select the most 
appropriate instructional setting fpr training each task to 'the Soldier's Manual 
Standard. (Qualification Training) * , % 

2. . Record the basis for each instructional setting selection. * 

• Purpose 

* ' - / V 

Ae purpose of this aid is to help you choose instructional settings (trailing locations) 
for tasks selected for training within each skill level. Due to advancements in instructional 
technology it 1/ often more cost-beneficial and efficient to train tasks in a non-institutional 
1 (extension) setting. This aid is designed' to help you icfentify^as many tasks as possible 
for which extension training is appropriate. 

• Product f' 

•> * 

This job aid will result in a listing of all critical tasks in which each task is assigned £g r 
training to one of the following instructional settings: 

*. * 

a. Institution (Resident schjpl training) _ \ 

b. Supervised On-the-job. Training (S0J17 * 1 

c. Self-study * \ v t 

i -» > 

This output will be especially useful in the preparation of the Commander's Manual. 

• Overview, of Major Steps in Selecting Instructional Setting 

.Step 1. Tasks selected fpr'training are categorized by skill level and duty position. 

.Step 2. Task performance data is obtained and recorded for eadh task. That is: 

, ^a. In- which* duty position is the task performed? 

„ - . b. What *percentage of soldiers perform the task? 
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Step 3. Irtitial assignment/ of tfye task to ^xrie of three instructional settings (institu- 
tion, supervised on-the-job training, or self-study) is made based *on the . 
answers to the following 14 questions: 

" ' S ' 

(1) Is task a common skill level task? ' . * -* 

(2) > Is task performed by a high percentage of" soldiers? * 

(3) Is task. performed in a similar manner in various duty positions 
, * and units? 

(4) t Is proficiency in task'' performance retained over time (that is, not 
'♦easily forgotten)? > , , 

(5) Does task rejquire considerable theoretical knowledge? 

/ ' .* . 

• (6) Must the task^be performed-immediately on entry to the job? . 

(7) Is the task a prerequisite for learning to/perform 6th£r school 
* trained tasks?. K * 

(8) Is trailing equipment a/d/or facilities only available at the school? , 

(9) Is the equipment required for individual teaming of the task in the , 
^ y { unit available .at most units? 

(10) Ar\personnel with the necessary expertise to conduct training of * 
the task available at ftiost units? , ^ 

(11) Do operational requirements at most units allow sufficient tinve for 
the soldier to receive training in the unit? * ' 

. (12) Can the task be learned with very little«supervision?" • 

* A , " *• ' " — 

(13) Does the soldier's schedule allow sufficient -time for independent 

study? ' ' v • ■ 

(14) . Can everything required for training (which is not already available 

in the unit) be included in the exportable training package at a cost 
competitive with school (institution) training? " ' 9 . 

Step 4. Administrative review and final "selection of instructional setting. 

. , * *i - 

In Step 3', tasks are assigned initially to one of the three instructional settings 
In this step each task is reviewed to determine if the initial assignment is 
still the best instructional setting on the basis of expert opinion. In the review 
of each task, questions of the type shown on the following displays are asked: 

• • • * 

(1) Have so many tasks been assigned to &OJT or Self -Study .that the^re is * * 

- not enough time to train, all the tasks before they must be performed? l « 
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. (2) Hav&.sb many tasks b#en assigned to^SO'JT that unit's can't handle 
* the load?^ , ' . " ' * 

{3) H&ve so^few tasks been assijgned to a particular^ setting that ^dminis . 
. trative costs outweigty the adv&ptages of training so few tasks^n this . * 
. instructional setting? % ' * 

* (4) Would resoijrce and time Constraints m the development erf new train- 
* - ing programs delay the onset of critically needed training? . •« 

'(5) Are there any other reasons why -theinitially selected instructional 
* setting should be changed? ^ 9 * 

. After reviewing eafch task and Asking questions similar to Xhosejust shown, 
any indicated changes in instructional setting' is made. The rationale 1 f6r ' 
each change is tt> be 'carefully documented. * , 

The ta&k listing with the final selections of instruction setting is .submitted 
to the supervisor for review and revision. *8 t * * v 
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DESCRIPTION OF INSTRUCTIONAL SETTINGS 



< 







^ Supervised On-The-Job 


>> 


Institution (INST) 


Training (SOJT) > 


' *Se\1 -£hidy ^* „ >. 


Training conducted at TRADOC 


Trairnng conducted* at the 


Training administered during 


resident schools and includes: 


soldier's unit 


individuals own time, usually 


OSUT *- 
PNCOC - 


One Station Unit 
Training » 
Primary Noncom- 




at the soldier's unit and includes: 
Self -teaching exportable 
packages (STEP) 
* Training Extension Courses' 
. - Job Performance) Aids 
-*-Study Guides 

Correspondence Courses 
Films, tapes, etc. • * 


BNCOC - 


missioned Officer 
Course * * 
Basic Noncom- 
missioned Officer 






^Course e 
/^vanced Noncom- 


* 


ANCOC - 








missioned Ofiifcef 
Course 


* 




Training always conducted under 


Training supervised by best 


Little supervision 'required 


supervision of qualified 


qualified NCOs in unit 




instructors' 
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■ ADVANTAGES AND .DISADVANTAGES OF 
THREE INSTRUCTIONAL 'SETTINGS* 



• Instructional Setting 


Advantages * 


Pisadvantacjes 


/ 

1 

? 

INSTITUTION 

N ' o 

e . *• 


V Usually best setting for teaming 
comrnpn skill level tasks or 
t^sks that are performed by 
large percentage of soldiers in 

' the MOS/skill level* ' 

Sophisticated training resource 
and expertise avaiftble 

* » 

_ 


Lack of real world environment 
"a 

High cosfcof soldier's housing • 
and travel 

Time is* spent away from job 
assignment - 

\ . 


SUPERVISED 
ONTHE-40B 
TRAINING 
< 

V 


£ffective for training tasks that 
can be learned faster or better 
hands-on experience 
* * 

Soldier contributes to unit's 1 
mission while learning 


May tie up unit's equipment and 
thus equipment may not be avail- 
able for operational use. 

May overburden supervisors, 

v Reduces time available^ in unit for 
operational requirements * 


/ 

SELF-STUDY * 

r , 

> 

4 . 


Effective' for training tasks 
which can be learned without 
' aft instructor or where little 
supervision is required • a 

Can be accomplished a*t * 
trairtee's convenience * 


If study occurs duri'rjg normal 
doty hours, this type of training 
may reduce time available in unit 
m for operational requirements 

May require soldier to devote 
considerable oft -duty time to 
study 



SOURCES OF INFORMATION FOR DETERMINING PERCENTAGE OF x 
SOLDIERS IN THE SKILL LEVEL WHO PERFORM EACH TASK 



CODAP Group Summary 

> 

CODAP data, when available, are excellent for determining the percent of soldiers 

within the skill level who perform each task. However, it is likely that not all tasks 

represented in th6 skill level will be represented on the CODAP Report. "Consequently, 

even when a CODAP Jleport is available, it will probably be necessary to supplement the 

data .front the Report with data from other sources. 

/ • 

c 

Field Survey \ 1 v 

Field Survey data are excellent fo* determining the percent of soldiers who perform 
each task. However, a field survey should only be conducted when CODAP data are not 
available (or badly out of date) and when there is sufficient time to conduct the purvey. 
Guidance f©r conducting a field survey can be found in Chapter III of this guide. 



Panel of Recent Job Incumbents 

This represents a fair source of information for determining the percent of soldiers 
within the skill level who perform eachKtask. See Chapter IV for guidance in establishing 
and using a panel of recent 'job incumbents. 



Panel of Subject Matter Experts 

Use this source only if none of the above 'sources are available. See Chapter IV 
for guidance in establishing and U6ing this type of panel. s 

Your Own Judgment 

As a last resort, you may have to, use your own judgment. - 

SOURCES OF INFORMATION FOR DETERMINING IF TRAINING 
REQUIREMENTS FOR THE TASK ARE ESSENTIALLY THE SAME • £ 

• REGARDLESS OF THE MISSION, EQUIPMENT ALLOCATION, 
GEOGRAPHICAL LOCATION, ETC., OF UNITS IW^WHICH THE 

.JOB INCUMBENT IS ASSIGNED 

— Job performance measures or task performance descriptions, developed in ISD 1.3 
Construct Job Performance Measures is an excellent source of equipment used in 
task performance. This will assist in determining whether equipment differences 
between units will have an effect *>n training requirements. \ 



— TOE/MTOE and TDA is anofher excellent source of information concerning 
equipment allocation in various units. ' . 

— Training Manuals antl supply bulletihs used in conjunction with TOE should he 
considered as a good source of information. ^ 

— Panel of recent job incumbents can provide good information for answering 
question 3. See Chapter IV for guidance in establishing and using this particular 
panel. * 

— Panel of subject matter experts. This also* is a good source. See Chapter IV for 
guidance in establishing and using a panel of subject matter experts. 

— Your own judgment. Use only as a last resort or in conjunction with other sources. 



EXAMPLES OF FACTORS TO CONSIDER* IN DETERMINING IF A 
TASK TAUGHT IN THE SCHOOL (INSTITUTION^ WILL STILL BE 
REMEMBERED BY THE TIME THE SOLDIER HAS TO PERFORM, 
THE TASK ON.THE JOB 



— Tasks for which the soldier has 'had previous civilian or military experience will 
usually ^e easily remembered (that is, driving a vehicle). 

— Tasks. which the soldier consid^ff important to remember will be better learned 
and more easily recalled. 

— Some tasks require frequent opportunities for practice in order to retain task 
proficiency. 

— In general, motor tasks (physical activities) are more easily remembered th&n 
mental tasks. * 

s 

— Tasks which involve short regular procedures are more easily remembered than 
those for which there is no clear cut procedure to follow. v 



SOURCES OF INFORMATION FOR DETERMINING IF THERE IS A 
CONSIDERABLE AMOUNT OF THEORY TO BE TAUGHT WITH THE ■ 
TASK UNDE RECONSIDERATION 

— Outpjjt from ISD 1.3 Construct Job Performance Measures or ISD 1.2 Conducting 
Ta§k Analysis provide a good squrce of information. Examine task descriptions 
to determine how the task is performed. This should provide an excellent insight 

* into the amount of theoretical content that will be required for training the task. f 

r * s 

— Training Manuals are an excellent source of information. 
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— Field survey of Job Supervisors is an excellent s6urce if time for survey is avail- 
able. See Chapter III for guidance) in conducting a field survey. 

— Panel of Recent Job Supervisors. Fair Source. See Chapter IV for guidance 
in establishing panel. . 1 e * r - - * , 

sources of Information for determining if the task 
MUST BE performed immediately on entr? to the jop 

— Field Svtfvey of Job Supervisors and/or Incumbents. Excellent source when time 
'for survey is available and if infor!matioh»is not available through COD^P- See 
Chapter III for guidance in condijfcting the survey. k 

. — Panel of Recent Job Incumbents.! Good Source. See Chapter IV for guidance in 
selecting panel. j v/ 

— Your own judgment. Use only if all other sources are unavailable. 



SOURCES OF INFORMATION FOR DETERMINING IF^THE TASK IS A 
PREREQUISITE FOR LEARNING AND/OR PERFORMING OTHER SCHOOL 
TRAINED TASKS 

— .Check Training Manuals, task description etc., to determine training equipment 
requirements. Survey fielcf supervisors to determine if training equipment is 
available in field units. ' » 

— Panel of Recent Job Supervisors. Use in conjunction with Training Manuals, 

• Job Performance Measures, etc. See Chapter IV for guidance in selecting panel. 



SOURCES OF INFORMATION FOR DETERMINING IF EQUIPMENT 
REQUIRED FOR INDIVIDUAL TRAINING OF THE TASK IN THE UNIT 
IS AVAILABLE AT MOST UNITS 

— Field Survey of Supervisors is an excellent source of information if time for sur- - 

vey is available. See Chapter IV for guidance in conducting a survey. 

*■ * ' 

— Panel of Refcent Job Supervisors is a good source of information. See Chapter IV 
for guidance in selecting panel. 

— Training Manuals to determine (equipment requirement followed by review of 
TOE/MTQE"or TDA for appropriate units. Good source of information/ 
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SOURCES OF INFORMATION FOR DETERMINING IF PERSONNEL WITH 
THE NECESSARY EXPERTISE ARE AVAILABLE AT MOST UNITS TO- 
CONDUCT THE TRAINING FOR THE TASK > " 

— Field Survey of Supervisors is ah excellent source of information if time for survey 
is available. See Chapter III for guidance in conducting a field survey. 

— Panel of Recent Job Supervisors. Good Source. See Chapter IV for guidance 
r in convening the panel. 

— Your own judgment. Use only if other sources are not available. 

<C ' - 

SOURCES OF INFORMATION FOR DETERMINING IF OPERATIONAL . 
REQUIREMENTS AT MOST UNITS, ALLOW SUFFICIENT TIME FOR 
THE SOLDIER TO BE TRAINED IN THE UNIT '? , 

— Field Survey of Supervisors is an excellent source of this information. See 
Chapter III for guidance in conducting a field survey. 

— Panel of Recent Job Supervisors. Good source. See Chapter IV for guidance 
in selecting panel. - 

SOURCES OF INFORMATION FOR%)ETERMINING IF TASK CAN BE 
LEARNED WITH VERY LITTLE- SUPERVISION 

— Output from ISD 1.2 JSelect Task/FunctrtJhs. Task learning difficulty should have 
been established on a rating scale of 1 to 7 and will therefore be an excellent 
indication, of the amount of supervision required. 

— Field Survey of Job Supervisors. Excellent source if time for survey is available. 
See Chapter III for guidance in conducting field survey. 

— Panel of Recent Job Supervisors. Good source of information. See Chapter IV 
for guidance in selecting panel. 

— Your own judgment. Use only if other sources are not available. 



SOURCES OF INFORMATION FOR DETERMINING IF THE SOLDIER'S 
SCHEDULE ALLOWS SUFFICIENT TIME FOR INDEPENDENT STUDY 

— Field Survey of Job Incumbents and Job Supervisors. Excellent source especially 
when a comparison is made between the two sources. See Chapter III -for 
guidance in conducting a field survey. ' 

— Panel of Recent Job Incumbents and Recent Job Supervisors. Good source. See 
'Chapter IV for guidance in -selecting panels, 

— Your own judgment/ Use only Jf other sources are not available. 
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SOURCES OF INFORMATION FOR DETERMINING IF EVERYTHING ; 
REQUIRED FOR TRAILING (WHICH IS NOT ALREADY AVAILABLE, 
IN THE FIELD) CAN BE INCLUDED IN A TRAINING PACKAGE AND IS IT 
INEXPENSIVELY EXPORTABLE - 

• • * 

— Check with Course Development Personnel 

— Panel of Subject Matter Experts. Fair source. See Chapter IV for guidance jn 
selecting, panel. 

,N \3 — Your own judgment. Use only if other sources are not available. 

FACTORS TO CONSIDER WHEN RE-EXAMINING THE TASKS 
INITIALLY ASSIGNED TO AN INSTITUTIONAL INSTRUCTIONAL" 
SETTING ' 

• For each task initially assigned to the Institution consider any reason why the 
task should NOT be trained in an institutional setting. Following are examples 
of questions you might want to ask as you review each task: 

* 

% — Do feedback from the field or SQT |j$£ults indicate that an Institutional setting 
has proven ineffective for any of these tasks? , \ 

— Are appropriate cues or stimuli not available in the school for any task? 

— Are skilled instructors available for teaching the task in the school? 

— Is' time and money available for training the task in the School? 

— Are there any job factors unique to this MOS which would cause you to change 
this instructional setting? 

— Will new equipment/simulators soon to b^vailablp cause you to change this 
setting? 

You may have other reasons why the 'task should not be taught at the .insti- 
tution. Ccnsidjer each reason carefully. 

FACTORS TO CONSIDER WHEN RE-EXAMINING THE TASKS INITIALLY 
ASSIGNED TO AN SOJT INSTRUCTIONAL SETTING 

• For each task initially assigned to S0JT considef any reason why the' task should 
NOT be trained in a-SOJTf setting-: Foilowing are examples of questions yau N 
might want to .ask as you revie^ each task. - ' * 

— Do feedback frtfm the field. or SQT results indicate that a SOJ^etting has 
proven ineffective fox- any of' these tasks?- . * • | 



— Is the environment too hazardous for training this t&sk in the field? For 
•example, would errors macTe during training in a field environment be critical 

1 to the student, to others, or to equipment? 

— Will new equipment/simulators soon be available which would cause you to 
change the setting? * \ 

— Do ^percentage performing" figures indicate that fewer soldiers aire performing 9 
this task than should be?. If so, does the percent which should be performing 
meet the criterion for high task performance and therefore consideration for 
Institutional trainttig? • - 

— *Are„ there ai\y job factors unique to this MOS which would cause you to change 
this instructional^ setting? s % v 

You may have other reasons why the task should not be taught by SOJT. 
Consider each reason carefully. 



FACTORS TO CONSIDER WHEN RE-EXAMINING THE TASKS INITIALLY 

^ASSIGNED TO A'SfLF-STUDY INSTRUCTIONAL SETTING 

& ,/ - 

' Y 

• Ror each .task initially assigned to Self-Staidy consider any reason wh# the task 
„ -should NOT be trained in a Self-Study setting. Following are examples of ques- 
< tions you might want to ask as you review each task: 



:i2 



— Do feedback from the field br SQT results indicate that a Self -Study setting 
has proven ineffective for any of these tasks? 

C f 

— Will new equiprfient/simulators soon be available which would cause you to 
change the setting? 

— Do "percentage performing" figures indicate that fewer soldiers are performing 
this task than should be? If so, does the percent which should be performing 
meet the criterion for high task performance and therefore consideration for 
Institutional training? 



— Would the percent 4 performing figure change radically during mobilizatio 
Would the task then become a candidate for training in the Institution? 

_ ^ re there any job factors unique to this MOS which would cause you to change 
jthis instructional setting? 

You may have other reasons why the task should not' be taught by Self-Study. 
Consider each reason carefully. 
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Chapter III 

* ** • 






CONDUCTING FIELD SURVEYS 

> 


* to 




WHAT IS A FORMAL FIELD SURVEY? 


• 




A formal field survey is similar to the type of questionnaire survey conducted by 
the Army Occupational Survey Program, only it is conducted by instructional develop-* 
ment personnel within an Army service school. 






, * 

The use of questionnaires permits the job analysis team to make limited contact 
with large numbers of personnel; thus large amounts of information can be collected at 
a relatively low cost. Questionnaires can be mailed, to personnel who are asked to com- 
plete and return them, or they can be administered to groups of job incumbents and/or 
supervisors by local personnel who have the responsibility and authority to make sure all 
questionnaires are completed and returned. 






WHEN SHOULD FORMAL FIELD SURVEYS BE USED? 




* 


Whenever time does not allow you to access information from the Army Occupa- 
tional Survey Program, an alternate data collection method may be used. Fornjarfield 
surveys are suggested as an*alternate data source in the ISD Job Aids. Should yoif decide 
to conduct a formal field survey be sure to^txtain permission from*MILPERCEN in 
accordance with the guidelines in AR 600-46.1 

I V 






HOW IS A FIELD SURVEY QUESTIONNAIRE DESIGNED? 






<* * 

• Types of Questionnaires 





* 

•4 

4 


There are two types of questionnaires, the closed form and the open form. We 
suggest using the closed form, wtopn contains a list of possible items to be selected or 
blanks to be filled in with words or numbers. (In the open form questionnaire the 
respondent might be asked "list the ta§ks you now perform and rate the amount of time 
spent on each task." For an example of a closed form Questionnaire, see pages 14 and 15. 
This form has several advantages over the Gther alternative, the open form. It is likely 
to take a minimum amotint of time to fill out, thus increasing the -chances that it will 
be completed and returned. The process of tabulating and summ&nzfrig responses is 
simpler and less time consuming than with an open form questionnaire. Machine tabu- , 
lation and computer analysis of the completed forms arq practical when a large number 
of v questionnaires* is used. * , 






A properly designed closed form questionnaire is difficult to prepare.' The 
desjgner must be sure to include all possible responses that he expects from any of the 
soldiers Who' will complete the questionnaire. The items must be constructed so that 
they clearly communicate to the user 'exactly what the designer is -trying to ask. The 
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JOB INVENTORY 

JDUTY TASK LIST) 


AFSC 

921X0/922X0 ' 


Page 7 of 


44 Pages 


1 

( 


1 Check tasks you perform now { ^) i 












2. Add any tasks you-do now which are not listed . * ~* 








• 




3 In the "Time Spent" column, rate checked lY) tasks on time spent in your present job 








t 


• - * Time Spent Scale t 

1 - VERY MUCH BELOW AVERAGE 4 - ABOUT "AVERAGE 7 - VERY MUCH ABOVE ' 

2 - BELOW AVERAGE 5 - SLIGHTLY ABOVE AVERAGE AVERAGE 

3 - SLIGHTLY BELOW AVERAGE 6 - ABOVE AVERAGE 


* 


C. FITTING AND MAINTAINING LIFE RAFTS AND PRESERVERS 


CHECK IF 
DONE IN 
PRESENT 
JOB 


timespent 
ooingVme se 

tasks in 
present job 




I. 4 Clean lift preservers • 


1 


42 
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2 Own lift rafts 




43 








3 Condemn non-reperable hf« rafts or life preservers 


t * 


44 








4 Fit lift preservers 




45 


• 






5. Inspect life preservers 

3 










6. Inspect lift raft accessories m 
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• 


7. Inspect lift rafts * 
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• 






8. Inject or weight test carbon dioxide (CO2) cylinders or cartridges 




49 








9 Meka entries on or review Life Preserver Oste forms (AFTO Form 466} 
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1 


10. Make entries on or review Life Preserver Inspection Oate Record forms (AF TO Form 336) 


51 






J ' 


f ' 






• I 
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11. Make entries on or review Life Reft inspection Record forms (AFTO Form 1337} 




52 ' 








12 Meka entries on or review User Certification Label forms (AFTO Form 27) 
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13 Pack life preserve's 
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14. Pack life reft accessory containers * 55 


\ 






15. Pack life rafts J 

V 




56 






• 

* 


16. Perform functional tests of life preservers 


• 


57 * 




# 




17. Perform functional tests of life refts 
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18. Perform inflation tests of Irfe preservers 1 « 

'■ '■ < 
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■ * 

* 


19. Perform inflation tests of life refts 




GO 




' V 




20. Perform minor repairs to life preservers such as patching rips, teert. or holes 
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(Continued next page) „ t 
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JOB INVENTORY 
(DUTY TASK LIST) 



921X0/"922X0 



Page 8 of 44 Pages 



1 Chcc^ tasks you per form now | 

2 Add any tasks yoJ do now which are not listed 

3 In the "Time Spent" column rate checked l^^^tasks on time spent in your present |ob 



Time Spent Scale 



1 - VERY MUCH BELOW AVERAGE , 

2 - BELOW AVERAGE 

3^- SLIGHTLY BELOW AVERAGE 



4* ABOUT AVERAGE 
6 - SLIGHTLY ABOVE AVERAGE- 
t 6 - ABOVE AVERAGE 



7 - VERY MUCH ABOVE 
AVERAGE 



* 4 - - ■ 

C FITTING AND MAINTAINING LIFE RAFTS AND PRESERVERS {CONTINUED) 

. ^ i 


<* 




V 

CHECK If 
DONE IN 
PRE.5ENT 
JOB 


Time s^nt 
DOING THESE 

TASKS IN 
PRESENT JOB 


21 Rtfill C02CY'ino>rj 


* 

a*. 
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22. Send lift reft compressed gas cylinders to other agencies forest, refill, or inspection 
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23 Siort life rafts or Ufe preservers 
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IF A TASK THAT YOU PERFORM IS NOT LISTED ANYWHERE IN THE ENTIRE LIST. 
WRITE IT IN THE BLANK SPACES BELOW » J* 
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' . 
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greatest single problem with research methods is improperly worded questionnaires, as 
they produce faultv/data. If you intend to design your own questionnaires we suggest 

consulting the^ertfowing guides: v r - 

*j * ■.'.„' 

^ r Morsh, J.E. and Accher, W.B. Procedural guid$ for conducting occupational 

surveys in the United States 'Air force 9 (PRL-TR-67-11). Lackland Air Force « 
Base, Texas: Personnel Research, Laboratory, Aerospace v Medical Division, J 
Air Foijy Systems Command, September 1967. 

Jacobs, T.O. Developing questionnaire items: how to do It well Human 
Resources Research Organization (HumRRO), 300 North Washington Street, 
Alexandria, Virginia 22314'. « . - ' $ 

WHAT^HOULD BE INCLUDED ON THE QUESTIONNAIRE? * . - 

c / 

The detaijs of. the forms you use will be/cfetermined by:* 

1. how you will tabulate and summarize the results, and 

2. what information you wish to collect., , 

\ How you^vill tabulate and summarize results will be determined by whether you 
have available a computer and other automated, data handling equipmenj and*>y the 
number of people surveyed.. To detertnine what information you wish to collect, you - 
. should consider the total data tequirement for the training development process. These . 
data requirements should befpredetermined in the job Analysis plan so that as, much 
information as possible can l?e obtained in a* single qqestionnaire survey effort. 



One note of caution about the design of your questionnaire is that you should keep 
the questionnaire as short as practical. In general, ^the forms should be designed so they 
can be completed in two hours or less. One way^ydu pan save time on a complex task 
inventory is to list all tasks under their appropriate duty 'position title. -This will permit 
the soldier to rapidly scan groups of tasks that slie/he does nSt perform and then proceed 
the next duty position. * 

WHAT TYPE OF INSTRUCTIONS FOR COMPLETING AND 
ADMINISTERING THE QUESTIONNAIRES SHOULD BE PREPARED? 

After the formal field survey questionnaire's have been written, the instructions for 
completing and administering the questionnaires should be prepared. These instructions 
should include: w \ M * 

• For the user — an introduction explainingthe purpose and lmpor- 

- tance of participating in the field survey. 

— general instructions explaining howthe questionnaire 
is to b.e completed. 

• For the project officer — general instructions regarding his/her responsibilities. 

(When questionnaire fs — specific instructions for administering the question- 
' - notfself-administered) ti'dnnah;e in a controlled environment. 

For examples bf these types of instructions, see-pages 17-23? ' ' % 
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TO: USER OF QUESTIONNAIRE 



INTRODUCTION 



TO THE NONCOMMISSIONED OFFICER: 

This questionnaire is, part of field survey designed to identify tasks for military 
police training. Its specific, purpose is to obfain from you, the Noncommissioned Officer, 
information on task criticality and frequency of performance. Feedback gained from 
this questionnaire will play a major part in pedesigning tl)e Noncommissioned Officer 
Advanced Education System. The ultimate gSal is to design training. so that it reflects 
what we have learned from you in the field. This goal ^possible only with your full 
cooperation. Consider each task listed in this questionnaire carefully and give your best 
response.. Your contribution is essential to a successful survey. 



\ 
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TO: USER-OF QUESTIONNAIRE, 



2. 

V 

\ 

3. 



GENERAL INpRUCTIONS 



1.' Complete thiy'survey questionnaire within the ( time. specified by your unit project 
officer and return it to him/her^upon completion. • , < 



Because- instructions for completing each part of this survey questionnaire are 
different, read all instructions carefully. 

Part II requires that you supply- biographical information. This information will be 
used to correlate feedback received from the field, Print all answers in the spaces 
provided on the appropriate survey questionnaire page. 



4. In the upper right corner of each page of Part III, Task Inventory, of this survey 
questionnaire is a BOOKLET NUMBER block/ Immediately to the left of this 
block is the individual booklet number. Print the individual booklet aumber in the 
BOOKLET NUMBER block on each page of the Task Inventory as demonstrated in 
the example. 



EXAMPLE: 



(000345) 









<* 






\ 




BOOKLET NUMBER 




■-0. 


1 2 


1 4 5 


6 




9 




. 0 


U 2 


3 m 5* 


6' 


1 t 8 


9 




0 


1 - 2 


0 4 H 


6 


7 8 


9 



5. Part III, Task Inventory, is divided into nine (9) separate sections (Sections I-IX^- 
The content of these sections con cqyjis. tasks you may perform in your present duty 
assignnfent. Yoy are asked to rate each task in accordance with three criteria - 
frequency of task performance, immediacy of task performance, and importance of 
task to mission success. 

Base all selections on your experience in your present duty assignment. 



Column A requires that you rate how often you perform each task on a scale 
from one to four. The criterion for this rating is tlie frequency of fask per- 
formance. Those tasks performed most frequently will noppajiy ht rated four 
while thpse" tasks not performed at all will be rated og 
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b. Column B requires that you determine how soon you must be capable of per- 
forming each task' after reporting to your present duty assignment. The'cri- t 
terion for this rating is ttte immediacy of task performance. Of the four' 
possible responses, Select the one most nearly describing your requirements. 
Select response number four for those* tasks which you must be capable of 
'performing«*ri\mediately upon reporting for duty. Select response number one 
for those tasks which you pever perform. r • 

* <■ 

c. Column C requires that yqu desojjbe, in your opinion, how important each 

• task is to mission success. The .criterion for this rating is the importance of 
the task to the accomplishment of the unit mission. Those tasks that, in 
. your opinion, are most important to mission success will be rated four while 
those tasks that you consider least important will be rated one . , ' 

f After selecting, enter your responses for each task, using either a-pen'orpencil, in - 
the answer portion adjacent to the appropriate task. statement as demonstrated in.— 
the example. > , • • . „ . 



EXAMPLE : The task PREPARE CORRESPONDENCE , if rated as performed 

FREQUENTLY in Column A, identified as.mfis^ke capable of performing 
IMMEDIATELY in Column B, anc* determined by you to be MOST 
IMPORTANT in Column C, would -tye entered in the answer portion, as 
shown below. 



PREPARE CORRESPONDENCE 1 



3 • 4 



■ 



■ 



/ 




After each section of Part III, Task Inventory, is ? Write-In Section: These write-in 
sections are prbvided in order that you may comment on each task inventory section, 
or list any task(s) you think should be included in the Task Inventory. These sections- 
also allow you to comment on those tasks that you find are the most difficult for 
you to jgpxform. 
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TO: PROJECT .OFFICER 



•GENERAL INSTRUCTIONS FOR PROJECT OFFICER ^ 

% J * ' ' / w 

1. General . The Military Police School is currently involved in redesigning basic 

military police training to pr6duce military policemen better equipped to perform 
when they reach the unit. Tho^erophasis is toward training replacements in tasks* 
actually being performed in the field. The questionnaires in this packet are designed 
to identify thq§e tasks. *' ' * 

The care with which you, the project officer, administer the questionnaires will 
determine tf)e ac^urjjcy'pf field feedback and, consequently the 'success or failure 
i of -this project.' « , * 

2. ^ Survey Packet Contents . ' „ ■ 

a. Questionnaire 

b. Supervisor Questionnaire t 

c Project Officer Instructions * u * 

d. ' Answer Sheets for^Questionnaire 

e? Pencils for use on answer sheets. 

3. Responsibilities . 

a. ^nit Commander . The unit commander is requested to appoint a project officer 
, , and to monitor administration of the survjpy. " ■ , # 

b. , Project Officer . The project officer is Responsible for the confrol and handling 

of questionnaires, for the administration of the questionnaires, and for returning 
completed and unused questionnaires to the Military Police School. 9 

c. Questionnaire Administrator . The project officer may appoint -someone to 
administer the questionnaire, if necessary. Normally, however, it is recom- 
mended *that the project officer administer jthe questionnaire. 



4, • About the Questionnaires. 



a. Questionnaire . ^ This questionnaire is designed to identify* tasks. being performed 
by military .policemen in the field and the frequency with* which each task 

is performed. f * ** 

b. Supervisor Questionnaire . Tht supervisor questionnaire is programmed to pro- 
vide feedback on task cnticality, probability *of deficient performance, and the 
frequency* with which each task is performed. 



20 



26 



1 



5. Who Takes The Questionnaire. The project officer is responsible for selecting 
individuals to take the questionnaires (respondents) within their units. Those 
selected must meet the requirements listed below: 

a. The questionnaire respondent mus|t: / 

(1) Be in an M.P. duty assignment (actually performing MP. duties) 
> (2) Have been on the job at feast 90 days 

b. The respondent to the Supervisor Questionnaire must: 

(1) Command or supervise M.P. personnel 

(2) Have been in a command or supervisory position in the unit for 90 days 
(Assign questionnaires proportionately among officers and NCO's) 

6. Questionnaire Administration . 

a. Questionnaire . The questionnaire will be administered in a controlled environ- 
ment. Persons participating in the survey will be allowed, two hours to complete 

t % the questionnaire and will turn the questionnaire and answer sheets in to the 

questionnaire administrator prior to leaving the survey area. Individual question- 
naires and their accompanying answer sheets will be kept together. 

See attached item for the procedure to be followed in administering the 
s questionnaire. f 

b. Supervisor Questionnaire . Supervisors selected as respondents for this ques- 
tionnaire will be allowed to sign for the questionnaire and take it with them. 
They will complete the questionnaire and return it to the project officer within 
a time frame he/she specifies. This time frame must be compatible with the 
suspense date to the Military Police School. 

7. . Questionnaire Handling . . Questionnaires and answer sheets become FOR OFFICIAL 
USE ONLY when completed. For ease of' accounting, each quest ibnnaire and its 
accompanying answer sheets are numbered. All questionnaires must be returned to 
the Military Police School whether they are used or not. Instructions for returning 
the questionnaires to the Military Police School are contained in the basic letter. 

If you have any problems or questions, contact (NOTE: Grve^name or jiames, 
address, and telephoife number). 

! i : £ 2_ 

* v 1 

' ; • ^ 

' I* f * 
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TO: PROJECT OFFICER ADMINISTERING QUESTIONNAIRE IN A CONTROLLED 
ENVIRONMENT 



ADMINISTERING THE QUESTIONNAIRE . 

" f 

i 

^ 

A-l. Preparation. A classroom or training room equfipped with desks will provide the 
most ideal site for administering the questionnaire. Questionnaires, answer sheets, 
and two electrographic pencils should be issued to participants after everyone who 
is to take the questionnaire's arrived. This ensures that everyone starts together. 

2. Instructions . The questionnaire administrator will present the following instructions. 

a. "Is there anyone here who is not working in an M.P. duty position? Is there 
anyone here who has not been assigned to their present duties at least 90 days? 
If so, please leave at this time." 

b. "Will everyone at this time please read the first page in the questionnaire whicji 
has beeft issued to you." 

( Note to the administrator: It must be emphasized that your enthusiasm for 
this project or lack of it will be contagious. - ft is important that you demon- 
strate a positive, attitude to the participants. Allowtime for the first page to 
be read and underline the importance of the questionnaire* with the following 
statement.) J 

"I would like to stress the importance of this questionnaire. The Military 
Police School wants to design training to fit the job in the field. You are 
the bnly people who can tell them what they need to know. Please think . 
through each question and-give your best answers." 

c "Turn to Part I, Biographical Information, and answer questions. 1-13. 

Whep you have finished', lay your pencil on the desk so I will .know when 
to proceed to the ne^ step." * 

d. "Now read the instructions found in Part II." • ^ 

( Notg/to the administrator : Allow reasonable time for everyone to finish before 
mo\H0g to the next step.) ^ 

"Are there any questions?" x ■ « 

e. "As you read in the instructions, there are nine answer sheets accompanying 
yoiyr questionnaire. Take the answer sheets and number them one thrpugh 
njne to correspond to the first nine sections in Part III of the questionnaire. 
If you do^not have nine answer sheets, raise your hand— I have extra ones. 
Use a separate answer sheet for each section. Answer only the number of 

- questions listed in each and move to the next section and answer sheet. It 
Is not necessary to write your name, rank, the date, oj> course at the top 1 
of the answer sheet. Also, disregard the blocks marked score, grade, extra ' 
points, and social security number." * y 
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f. "Because of the size of this survey, these answer sheets will be read by 
machine. You must use the special pencils provided so that the machine can 
read the answers. When marking vfa*r answer, take care to fill the vertical 

- rectangle outlining the letter as sfiown by the example on page 4 of the 
instructions. Also, please be sure not to make any stray marks on the 
answer sheets. . Finally, do not fold the answer sheets." ' . 

g. "All answers must be based on ydftr experience in this your present unit. Do 
not call on experience in previous units. This means that if you do not per- 
form a particular task an your present unit, you must mark 'do not.perform 

this task' on your answer sheet." 

». * * 

h. "You may begin answering Part III. Remember Sectipn ten, the written 
section. When you fthish answering all X[ues£i6ns, insert your answer 
sheets into the questionnaire and turn tfeem in to me. You may leave 
when you are finished. Are there any questions?" 

A-3. Conclusion . After everyone has taken the questionnaire, ensure that all ques- 
tionnaires and answe^ sheets are accounted for. Collect the pencils provided so 
that they may be returned to the Military Police School along with th& question- 
naires and answer sheets. 



23 



29 



HOW IS A SURVEY SAMPLE SELECTED? 

You are now ready to select organizations and individuals to provide you with the m 
needed job data. The complexity, of the MOS, the availability find quality of published 
sources of job information, and the number of people in the particular job will determine 
how- much and what kinds of information you need to collect. This will strongly 
influence the^make-up and size of your sample. If the complexity and amount of required 
data are great, the number of organizations "and individuals interviewed will increase. As 
a general rule, your survey sample should be as large as possible. This is particularly true 
if you do not have ^personnel available with the ^responsibility and authority to assure that 
most of your questionnaires will be completed. You should make an attempt to obtain a 
sample that represents the distribution of individuals in the MOS according to command 
and skill level. Review of. personnel records, either by personnel employees, members of 
your job analysis team, or your field representatives whg will conduct the survey will be 
''required to objtain data upon which tQ c base choices for your survey. - 

• In selecting UNITS for sampling, you should select units that: 

1. have at least a small number of job holders and supervisors who do the 
particular -job to be arfalyzed. Preferably, you should choose some units 1 
that .have a relatively small number of job hqlders, and some that have 
large numbers. 



< 



# \ 2. are geographically and environmentally representative. ' ? < 

• In selecting INDIVIDUALS ^vithin the units" you should ^select a group made up 
; of individuals who: * * ' 

1. -perform and supeigrise the- job being analyzed. 

2. perform with average satisfactory proficiency. 

. . 3. are representative in terms of length of time on t)te jo*b, 
4. are representative in terms of training. 

-For cei^ain types of information yo'u'will also want to choose at least '-a few job holders or 
supervisors who are acknowledged experts at the job. . * 4.' ' 



HOW IS A QUESTIONNAIRE SURVEY CONDUCTED? / 

\ V > > ' 

Trial Rurr (Validation of Imtrumehts) * ' J * - * 

Before sending out the total number of questionnaires you intend using, you may 
wish' to send out a small number. This will permit yoii to check the initial results and 
possibly make sortie changes in your questionnaire^ or instructions. Then you will send * 
out what you hope will be^the total number of questionnaires required. 



• Group Administration 

* ' The ideal way to administer questionnaires is grtfup administration. Where the 
local responsible official and his/her assistants schedule the administration he/she should 
do the following: "* . _ 

— Make certain that only eligible individuals are seated in thfe administration area. 

— Read the administrative instructions. 

— Provide any necessary assistance in completing the questionnaires. 

— Return the completed questionnaires to the school. ; 

\ • Individual Administration « 

Often, particularly with individuals at remote stations, group administration is 
impractical. In these cases, it is sometimes effective to send the' questionnaires to a 
responsible officer and request that he/she return them by a reasonable suspense <^ate. 
Careful attention should be paid to the instructions for administration of self-administration. 
If your command has no authority to require that a suspense date be met, then you must 
either obtain the concurrence of a command with that authority, or be willing to accept 
a reduced percentage return. 

• Return of Questionnaires A 

How muciycbnfidence can you have in the validity of your questionnaire if you 
get less than a lOfk percent return? Less and less coivfidence can be expected with each 
reduction in the v percentage retitfnect. What can you do if you are not satisfied with the * 
percentage of returns of the questionnaires? We suggest you try the following: 

1) Sehd out more forms to different people and hope for better results. 

2) I^econtact some of the- first sample»and try to encourage them to return 
the questionnaires, 

3) Visit a*random sample of those who did not respond and compare their 
forced responses with the voluntary responses. Then you and your super- 
visors will have to decide how much chance you are willing to take that . 
the data you have received presents £ sufficiently accurate picture of ' the 
job as it really ^xists. 
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Chapter IV 
ORGANIZING PANELS OF EXPERTS 



WHAT IS A PANEL OF EXPERTS? 

With this method a group of personnel, selected for their experience and knowledge 
of the job, is brought together to confer about the required job analysis data. Panels 
may be njade up of one or more of the following types of members: 

1) Subject Matter Experts (SMEs) 

These are personnel found at your^chool who are acknowledged experts 
« in the tasks, duty positions or MOS you are analyzing. They may be 
found among instructors or ISD personnel who hold the MOS. SMEs 

fc may or may not have had recent field experience. 

y * • 

2) v Job Incumbents 



This group includes those who are presently holding the jobs/performing 
the tasks you are Interested in, or who have recently held the jobs. The 
. , more recent the better. More than^three years away from the field would - 
disqualify a potential panel member. 
r ' 

3) Job Supervisors* 

This group includes those who are presently or haveTecently (within the 
t past % three years) supervised 'soldiers in the jobs/tasks you are interested in. 

*- 1 \ - ■ 

Job incumbents and job supervisors may be found on the post where your school is ■ 
located^pr at other locations. Check the TOEs and TyAs to find out whfere personnel _ 
may be assigned. While personnel' assigned to your tyost are mo^t*coYfv£nierttly assembled^ 
they may no.t be completely representative of job inOUmbent^/Stipervisors serving in other, 
locations. Also, they probably haVe been heavily Burdened by school surVeys, panels,, etc:/ 
already due \o their proximity to the school. Therefore, personnel frp.rft other locations" 
may have^tcbe used.- \ . 

WHEN IS A PANEL USED? ~ - 



Panels, of experts are h .good alternate data source when: 

— CODAP is not available. > " 

— time and funds are inadequate -for conductfng.a field survey; 

— the type* of information required can be reliably "provided by a small group. 
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WHEN ARE SUBJECT MATTER EXPERTS USED? 

With this method, a group of highly experienced personnel is brought together to 
record and organize the required job analysislpata. This method is particularly useful in 
collecting 40b data on new jobs or on managerial and supervisory jobs where many of 
the most critical behaviors are not directly observable. Since the members of a panel of 
SMEs are experts m'the MOS bepg analy zed w t heir collective effort should be decisions 
about the freqnirements of the job. In general, their greatest effectiveness is in evaluating 
and making decisions about job data that have been collected from other sources by • 
other nrcans. 



WHEN ARE RECENT JC^ INCUMBENTS AND/OR SUPERVISORS USED? 

With this method, a group of job incumbents, job supervisors, or a combination of 
these, is brought together to provide information about their jobs. The primary- function 
of this~type of panel is to provide information about their jotjs, not to make decisiorfs. 
Another name for 4his type of panel is Consensus Group. 



HOW IS A PANEL ASSEMBLED? • ; . . , 

The panel is ajrelatively inexpensive and easy approach to .collecting data. Three to* 
seven persons is,the number recommended to make up the panel. Whefieyip? possible, you 
want panel members to be representative of different locations and types of Units' in the # 
field.** While many experts may Be available v^ithih your schoo] it 13 critical that their 
views b$ balanced by those of persons serving presently or recently in the field. If all of 
the panel members come from-sch^ols, there may be a' tendency fpr the outcome to reflect 
what exists in training rather than what actually exists on the job. 
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